3D ConvNets with Optical Flow Based Regularization

Video classification using 3D convolutional neural networks still lags behind models with simple classifiers on top of rich, hand-engineered, spatio-temporal features for a number of prominent action recognition datasets. Many of these hand-designed features are built on top of estimates of optical flow. Thus we propose an extension to the 3D convolutional neural network model that incorporates the underlying assumptions of optical flow as a form of regularization and analyze the effects of this extension on early-stage training of a 5-layer architecture on the UCF-101 Human Action Recognition dataset.